Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EnvironmentLogger Callback #1350

Merged
merged 16 commits into from
Aug 17, 2024
Merged

Add EnvironmentLogger Callback #1350

merged 16 commits into from
Aug 17, 2024

Conversation

josejg
Copy link
Contributor

@josejg josejg commented Jul 12, 2024

This is a new callback to simplify logging environment metadata for reproducibility purposes:

  • Git Commits for packages under workspace_dir, useful for mcli integrations
  • Package versions for core dependencies, e.g. transformers
  • Nvidia versions, including cuda, cuddn and nccl
  • Docker image and tag
  • Distributed info
  • System info
  • Gpu info

Main change is that GitPython now becomes a dependency. It is already a composer[dev] dependency. I was not sure if it was worth gating under a additional dependency

@josejg josejg requested review from a team as code owners July 12, 2024 00:11
@josejg
Copy link
Contributor Author

josejg commented Jul 12, 2024

I tested it with both MLflow and Wandb, where it produced the following

MLFlow

image

Wandb

image

Copy link
Collaborator

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be included in the state data and logged? @eracah

@josejg
Copy link
Contributor Author

josejg commented Jul 29, 2024

Pushed an update fixing the failing checks. Let me know if it's good to go or if there's anything that needs to be changed

Copy link
Contributor

@b-chu b-chu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments and questions. Agreed with Mihir, it seems useful to incorporate this as state dict metadata instead of relying on loggers.

llmfoundry/callbacks/envlogger.py Outdated Show resolved Hide resolved
llmfoundry/callbacks/envlogger.py Outdated Show resolved Hide resolved
llmfoundry/callbacks/envlogger.py Outdated Show resolved Hide resolved
setup.py Show resolved Hide resolved
llmfoundry/callbacks/envlogger.py Outdated Show resolved Hide resolved
llmfoundry/callbacks/envlogger.py Outdated Show resolved Hide resolved
llmfoundry/callbacks/envlogger.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@dakinggg dakinggg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets mark this as experimental for now. we may incorporate it into composer directly.

llmfoundry/callbacks/env_logging_callback.py Show resolved Hide resolved
llmfoundry/callbacks/env_logging_callback.py Outdated Show resolved Hide resolved
@josejg josejg enabled auto-merge (squash) August 17, 2024 06:31
@josejg josejg merged commit d014d75 into mosaicml:main Aug 17, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants